{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save and Restore\n",
    "In this post we are going to talk about how to save the parameters into the disk and restore the saved parameters from the disk. The savable/restorable paramters of the network are __Variables__ (i.e. weights and biases)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TLDR: \n",
    "\n",
    "To save and restore your variables, all you need to do is to call the `tf.train.Saver()` at the end of you graph.\n",
    "\n",
    "```python\n",
    "# create the graph\n",
    "X = tf.placeholder(..)\n",
    "Y = tf.placeholder(..)\n",
    "w = tf.get_variable(..)\n",
    "b = tf.get_variable(..)\n",
    "...\n",
    "loss = tf.losses.mean_squared_error(..)\n",
    "optimizer = tf.train.AdamOptimizer(..).minimize(loss)\n",
    "...\n",
    "\n",
    "saver = tf.train.Saver()\n",
    "```\n",
    "\n",
    "\n",
    "__In the train mode__, in the session we will initialize the variables and run our network. At the end of training, we will save the variables using `saver.save()`:\n",
    "\n",
    "```python\n",
    "# TRAIN\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    # train our model\n",
    "    for step in range(steps):\n",
    "        sess.run(optimizer)\n",
    "        ...\n",
    "    saved_path = saver.save(sess, './my-model', global_step=step)\n",
    "```\n",
    "\n",
    "This will create 3 files (`data`, `index`, `meta`) with a suffix of the step you saved your model.\n",
    "\n",
    "__In the test mode__, in the session we will restore the variables using `saver.restore()` and validate or test our model.\n",
    "\n",
    "```python\n",
    "# TEST\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, './my-model')\n",
    "    ...\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Import the required libraries:\n",
    "\n",
    "We will start with importing the required Python libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#imports\n",
    "import tensorflow as tf\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Save and Restore Two Variables:\n",
    "### 1.1 Save:\n",
    "We will start with saving and restoring two variables in TensorFlow. We will create a graph with two variables. Let's create two variables `a = [3 3]` and `b = [5 5 5]`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create variables a and b\n",
    "a = tf.get_variable(\"A\", initializer=tf.constant(3, shape=[2]))\n",
    "b = tf.get_variable(\"B\", initializer=tf.constant(5, shape=[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the __lower__case letter as python name and __UPPER__case letter as TensorFlow name. It will be important when we want to import the graph in restoring the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Recall from the [Tensor Types Tutorial](https://github.com/easy-tensorflow/easy-tensorflow/blob/master/1_TensorFlow_Basics/Tutorials/2_Tensor_Types.ipynb):__ Variables need to be initialized before being used. To do so, we have to invoke a __variable initializer operation__ and run the operation on the session. This is the easiest way to initialize variables which initializes all variables at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize all of the variables\n",
    "init_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, on the session, we can initialize the variables and run the to see the values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a =  [3 3]\n",
      "b =  [5 5 5]\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # initialize all of the variables in the session\n",
    "    sess.run(init_op)\n",
    "    # run the session to get the value of the variable\n",
    "    a_out, b_out = sess.run([a, b])\n",
    "    print('a = ', a_out)\n",
    "    print('b = ', b_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Important Note:__ All of the variables exist in the scope of the session. So, after the session is closed, we will loose the variable. \n",
    "\n",
    "In order to save the variable, we will call the saver function using `tf.train.Saver()` in our graph. This function will find all the variables in the graph. We can see the list of all variables in `_var_list`. Let's create a `saver` object and take a look at the `_var_list` in the object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Var 0: <tf.Variable 'A:0' shape=(2,) dtype=int32_ref>\n",
      "Var 1: <tf.Variable 'B:0' shape=(3,) dtype=int32_ref>\n"
     ]
    }
   ],
   "source": [
    "# create saver object\n",
    "saver = tf.train.Saver()\n",
    "for i, var in enumerate(saver._var_list):\n",
    "    print('Var {}: {}'.format(i, var))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, our graph consists of two variables that listed above.\n",
    "\n",
    "__Important Note__: Notice the `:0` at the end of the variable name. For more about tensor naming check [here](https://stackoverflow.com/questions/36150834/how-does-tensorflow-name-tensors).\n",
    "\n",
    "Now that the saver object is created in the graph, in the session, we can call the `saver.save()` function to save the variables in the disk. We have to pass the created session (`sess`) and the path to the file that we want to save the variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model saved in ./saved_variable\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # initialize all of the variables in the session\n",
    "    sess.run(init_op)\n",
    "    \n",
    "    # save the variable in the disk\n",
    "    saved_path = saver.save(sess, './saved_variable')\n",
    "    print('model saved in {}'.format(saved_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check your working directory, you will notice that 3  new files have been created with the name `saved_variable` in them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saved_variable.data-00000-of-00001\n",
      "saved_variable.index\n",
      "saved_variable.meta\n"
     ]
    }
   ],
   "source": [
    "for file in os.listdir('.'):\n",
    "    if 'saved_variable' in file:\n",
    "        print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__.data:__ Contains variable values\n",
    "\n",
    "__.meta:__ Contains graph structure\n",
    "\n",
    "__.index:__ Identifies checkpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Restore:\n",
    "Now that all the things that you need is saved in the disk, you can load your saved variables in the session using `saver.restore()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./saved_variable\n",
      "a =  [3 3]\n",
      "b =  [5 5 5]\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # restore the saved vairable\n",
    "    saver.restore(sess, './saved_variable')\n",
    "    # print the loaded variable\n",
    "    a_out, b_out = sess.run([a, b])\n",
    "    print('a = ', a_out)\n",
    "    print('b = ', b_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that this time we did not initialize the variables in our session. Instead, we restored them from the disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Important Note:__ In order to restore the parameters, the graph should be defined. Since we defined the graph in top, we didn't have a problem restoring the parameters. But what happens if we have not loaded the graph?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./saved_variable\n",
      "The Session graph is empty.  Add operations to the graph before calling run().\n"
     ]
    }
   ],
   "source": [
    "# delete the current graph\n",
    "tf.reset_default_graph()\n",
    "try:\n",
    "    with tf.Session() as sess:\n",
    "        # restore the saved vairable\n",
    "        saver.restore(sess, './saved_variable')\n",
    "        # print the loaded variable\n",
    "        a_out, b_out = sess.run([a, b])\n",
    "        print('a = ', a_out)\n",
    "        print('b = ', b_out)\n",
    "except Exception as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can define the graph in two ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.1. Define the graph from scratch and then run the session:\n",
    "This way is simple if you have your graph. So, what you can  do is to create the graph and then restore your variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./saved_variable\n",
      "a =  [3 3]\n",
      "b =  [5 5 5]\n"
     ]
    }
   ],
   "source": [
    "# delete the current graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# create a new graph\n",
    "# create variables a and b\n",
    "a = tf.get_variable(\"A\", initializer=tf.constant(3, shape=[2]))\n",
    "b = tf.get_variable(\"B\", initializer=tf.constant(5, shape=[3]))\n",
    "\n",
    "# initialize all of the variables\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# create saver object\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # restore the saved vairable\n",
    "    saver.restore(sess, './saved_variable')\n",
    "    # print the loaded variable\n",
    "    a_out, b_out = sess.run([a, b])\n",
    "    print('a = ', a_out)\n",
    "    print('b = ', b_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that the graph should be exactly like the one that you saved. ButwWhat if we do not know the exact graph and we are using someone else's pre-trained model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2.2. Restore the graph from `.meta` file.\n",
    "\n",
    "When we save the variables, it creates a `.meta` file. This file contains the graph structure. Therefore, we can import the meta graph using `tf.train.import_meta_graph()` and restore the values of the graph. Let's import the graph and see all tensors in the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Const\n",
      "A\n",
      "A/Assign\n",
      "A/read\n",
      "Const_1\n",
      "B\n",
      "B/Assign\n",
      "B/read\n",
      "init\n",
      "save/Const\n",
      "save/SaveV2/tensor_names\n",
      "save/SaveV2/shape_and_slices\n",
      "save/SaveV2\n",
      "save/control_dependency\n",
      "save/RestoreV2/tensor_names\n",
      "save/RestoreV2/shape_and_slices\n",
      "save/RestoreV2\n",
      "save/Assign\n",
      "save/RestoreV2_1/tensor_names\n",
      "save/RestoreV2_1/shape_and_slices\n",
      "save/RestoreV2_1\n",
      "save/Assign_1\n",
      "save/restore_all\n"
     ]
    }
   ],
   "source": [
    "# delete the current graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# import the graph from the file\n",
    "imported_graph = tf.train.import_meta_graph('saved_variable.meta')\n",
    "\n",
    "# list all the tensors in the graph\n",
    "for tensor in tf.get_default_graph().get_operations():\n",
    "    print (tensor.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you recall from section 1.1, we defined the python names with __lower__case letters and in TensorFlow names with __UPPER__case letters. You can see that what we have here are the __UPPER__case letter variables. It means that `tf.train.Saver()` saves the variables with the TensorFlow name. Now that we have the imported graph, and we know that we are interested in `A` and `B` tensors, we can restore the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./saved_variable\n",
      "a =  [3 3]\n",
      "b =  [5 5 5]\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # restore the saved vairable\n",
    "    imported_graph.restore(sess, './saved_variable')\n",
    "    # print the loaded variable\n",
    "    a_out, b_out = sess.run(['A:0','B:0'])\n",
    "    print('a = ', a_out)\n",
    "    print('b = ', b_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Important Note:__ Notice that in `sess.run()` we provided  the TensorFlow name of the tensors `'A:0'` and `'B:0'` instead of `a` and `b`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Save and Restore Variables of a Sample Linear Model:\n",
    "Now that we have learnt how to save and restore parameters, we can write a simple model and try to save and restore the __weights__ and __biases__ in this network.\n",
    "\n",
    "We will build a simple linear model. If you do not know about the linear model, check our [Linear Classifier Tutorial](https://github.com/easy-tensorflow/easy-tensorflow/tree/master/2_Linear_Classifier)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# delete the current graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Data Dimensions\n",
    "img_h = img_w = 28              # MNIST images are 28x28\n",
    "img_size_flat = img_h * img_w   # 28x28=784, the total number of pixels\n",
    "n_classes = 10                  # Number of classes, one class per digit\n",
    "\n",
    "# Load MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "data = input_data.read_data_sets(\"MNIST/\", one_hot=True)\n",
    "\n",
    "# Hyper-parameters\n",
    "learning_rate = 0.001   # The optimization initial learning rate\n",
    "batch_size = 100        # Training batch size\n",
    "num_steps = 1000         # Total number of training steps\n",
    "\n",
    "# Placeholders for inputs (x), outputs(y)\n",
    "x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X')\n",
    "y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y')\n",
    "\n",
    "W = tf.get_variable('W',\n",
    "                    dtype=tf.float32,\n",
    "                    shape=[img_size_flat, n_classes],\n",
    "                    initializer=tf.truncated_normal_initializer(stddev=0.01))\n",
    "b = tf.get_variable('b',\n",
    "                    dtype=tf.float32,\n",
    "                    initializer=tf.constant(0., shape=[n_classes], dtype=tf.float32))\n",
    "\n",
    "# Calculate the output logits as: output_logits = W*x + b\n",
    "output_logits = tf.matmul(x, W) + b\n",
    "# Convert logits to probabilities\n",
    "y_pred = tf.nn.softmax(output_logits)\n",
    "\n",
    "# Define the loss function, optimizer, and accuracy\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y, logits=output_logits), name='loss')\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss)\n",
    "correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred')\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of graph, will call the `tf.train.Saver()` to save all the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create saver object\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run the model and save the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------\n",
      "Validation loss: 0.32, Validation accuracy: 91.4%\n",
      "---------------------------------------------------------\n",
      "model saved in ./linear_model\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    for i in range(num_steps):\n",
    "        # Get a batch of training examples and their corresponding labels.\n",
    "        x_batch, y_true_batch = data.train.next_batch(batch_size)\n",
    "\n",
    "        # Put the batch into a dict to be fed into the placeholders\n",
    "        feed_dict_train = {x: x_batch, y: y_true_batch}\n",
    "        sess.run(optimizer, feed_dict=feed_dict_train)\n",
    "\n",
    "    feed_dict_valid = {x: data.validation.images, y: data.validation.labels}\n",
    "    loss_test, acc_test = sess.run([loss, accuracy], feed_dict=feed_dict_valid)\n",
    "    print('---------------------------------------------------------')\n",
    "    print(\"Validation loss: {0:.2f}, Validation accuracy: {1:.01%}\".format(loss_test, acc_test))\n",
    "    print('---------------------------------------------------------')\n",
    "\n",
    "    # save the variable in the disk\n",
    "    saved_path = saver.save(sess, './linear_model')\n",
    "    print('model saved in {}'.format(saved_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that the model is saved in `./linear_model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "linear_model.data-00000-of-00001\n",
      "linear_model.index\n",
      "linear_model.meta\n"
     ]
    }
   ],
   "source": [
    "for file in os.listdir('.'):\n",
    "    if 'linear_model' in file:\n",
    "        print(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's restore the model and pull out the trained variables. at this time, the garph still exists in the memory. So, we can restore it and evaluate the network on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./linear_model\n",
      "---------------------------------------------------------\n",
      "Test loss: 0.32, test accuracy: 91.2%\n",
      "---------------------------------------------------------\n",
      "\n",
      "W =  [[-0.00795777  0.00179533 -0.01632378 ...  0.01127853 -0.0165657\n",
      "  -0.01905626]\n",
      " [-0.00036186 -0.01072511  0.01412622 ... -0.01868354  0.0091197\n",
      "  -0.01899963]\n",
      " [ 0.01169185  0.00378057 -0.00452419 ...  0.01239008  0.00784524\n",
      "  -0.00559599]\n",
      " ...\n",
      " [ 0.00589321 -0.01055128  0.00599118 ... -0.0021277   0.00593736\n",
      "   0.00538401]\n",
      " [ 0.0050825   0.00028797 -0.00596465 ... -0.00036289  0.00454178\n",
      "  -0.00049127]\n",
      " [-0.00797494 -0.00345959  0.002974   ...  0.01462011 -0.00961047\n",
      "   0.00482459]]\n",
      "b =  [-0.13907638  0.24205    -0.04328492 -0.09594144  0.0636861   0.19795758\n",
      " -0.03274642  0.12884071 -0.2828767  -0.04605722]\n"
     ]
    }
   ],
   "source": [
    "# Test the network after training\n",
    "\n",
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # restore the saved vairable\n",
    "    saver.restore(sess, './linear_model')\n",
    "    \n",
    "    # Accuracy\n",
    "    feed_dict_test = {x: data.test.images, y: data.test.labels}\n",
    "    loss_test, acc_test = sess.run([loss, accuracy], feed_dict=feed_dict_test)\n",
    "    print('---------------------------------------------------------')\n",
    "    print(\"Test loss: {0:.2f}, test accuracy: {1:.01%}\".format(loss_test, acc_test))\n",
    "    print('---------------------------------------------------------')\n",
    "    print()\n",
    "\n",
    "    \n",
    "    # print the loaded variable\n",
    "    weight, bias = sess.run(['W:0','b:0'])\n",
    "    print('W = ', weight)\n",
    "    print('b = ', bias)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from __Section 1.2__, if we do not have the graph, we can restore the values of the graph using `tf.train.import_meta_graph()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X\n",
      "Y\n",
      "W/Initializer/truncated_normal/shape\n",
      "W/Initializer/truncated_normal/mean\n",
      "W/Initializer/truncated_normal/stddev\n",
      "W/Initializer/truncated_normal/TruncatedNormal\n",
      "W/Initializer/truncated_normal/mul\n",
      "W/Initializer/truncated_normal\n",
      "W\n",
      "W/Assign\n",
      "W/read\n",
      "Const\n",
      "b\n",
      "b/Assign\n",
      "b/read\n",
      "MatMul\n",
      "add\n",
      "Softmax\n",
      "softmax_cross_entropy_with_logits/Rank\n",
      "softmax_cross_entropy_with_logits/Shape\n",
      "softmax_cross_entropy_with_logits/Rank_1\n",
      "softmax_cross_entropy_with_logits/Shape_1\n",
      "softmax_cross_entropy_with_logits/Sub/y\n",
      "softmax_cross_entropy_with_logits/Sub\n",
      "softmax_cross_entropy_with_logits/Slice/begin\n",
      "softmax_cross_entropy_with_logits/Slice/size\n",
      "softmax_cross_entropy_with_logits/Slice\n",
      "softmax_cross_entropy_with_logits/concat/values_0\n",
      "softmax_cross_entropy_with_logits/concat/axis\n",
      "softmax_cross_entropy_with_logits/concat\n",
      "softmax_cross_entropy_with_logits/Reshape\n",
      "softmax_cross_entropy_with_logits/Rank_2\n",
      "softmax_cross_entropy_with_logits/Shape_2\n",
      "softmax_cross_entropy_with_logits/Sub_1/y\n",
      "softmax_cross_entropy_with_logits/Sub_1\n",
      "softmax_cross_entropy_with_logits/Slice_1/begin\n",
      "softmax_cross_entropy_with_logits/Slice_1/size\n",
      "softmax_cross_entropy_with_logits/Slice_1\n",
      "softmax_cross_entropy_with_logits/concat_1/values_0\n",
      "softmax_cross_entropy_with_logits/concat_1/axis\n",
      "softmax_cross_entropy_with_logits/concat_1\n",
      "softmax_cross_entropy_with_logits/Reshape_1\n",
      "softmax_cross_entropy_with_logits\n",
      "softmax_cross_entropy_with_logits/Sub_2/y\n",
      "softmax_cross_entropy_with_logits/Sub_2\n",
      "softmax_cross_entropy_with_logits/Slice_2/begin\n",
      "softmax_cross_entropy_with_logits/Slice_2/size\n",
      "softmax_cross_entropy_with_logits/Slice_2\n",
      "softmax_cross_entropy_with_logits/Reshape_2\n",
      "Const_1\n",
      "loss\n",
      "gradients/Shape\n",
      "gradients/grad_ys_0\n",
      "gradients/Fill\n",
      "gradients/loss_grad/Reshape/shape\n",
      "gradients/loss_grad/Reshape\n",
      "gradients/loss_grad/Shape\n",
      "gradients/loss_grad/Tile\n",
      "gradients/loss_grad/Shape_1\n",
      "gradients/loss_grad/Shape_2\n",
      "gradients/loss_grad/Const\n",
      "gradients/loss_grad/Prod\n",
      "gradients/loss_grad/Const_1\n",
      "gradients/loss_grad/Prod_1\n",
      "gradients/loss_grad/Maximum/y\n",
      "gradients/loss_grad/Maximum\n",
      "gradients/loss_grad/floordiv\n",
      "gradients/loss_grad/Cast\n",
      "gradients/loss_grad/truediv\n",
      "gradients/softmax_cross_entropy_with_logits/Reshape_2_grad/Shape\n",
      "gradients/softmax_cross_entropy_with_logits/Reshape_2_grad/Reshape\n",
      "gradients/zeros_like\n",
      "gradients/softmax_cross_entropy_with_logits_grad/ExpandDims/dim\n",
      "gradients/softmax_cross_entropy_with_logits_grad/ExpandDims\n",
      "gradients/softmax_cross_entropy_with_logits_grad/mul\n",
      "gradients/softmax_cross_entropy_with_logits_grad/LogSoftmax\n",
      "gradients/softmax_cross_entropy_with_logits_grad/Neg\n",
      "gradients/softmax_cross_entropy_with_logits_grad/ExpandDims_1/dim\n",
      "gradients/softmax_cross_entropy_with_logits_grad/ExpandDims_1\n",
      "gradients/softmax_cross_entropy_with_logits_grad/mul_1\n",
      "gradients/softmax_cross_entropy_with_logits_grad/tuple/group_deps\n",
      "gradients/softmax_cross_entropy_with_logits_grad/tuple/control_dependency\n",
      "gradients/softmax_cross_entropy_with_logits_grad/tuple/control_dependency_1\n",
      "gradients/softmax_cross_entropy_with_logits/Reshape_grad/Shape\n",
      "gradients/softmax_cross_entropy_with_logits/Reshape_grad/Reshape\n",
      "gradients/add_grad/Shape\n",
      "gradients/add_grad/Shape_1\n",
      "gradients/add_grad/BroadcastGradientArgs\n",
      "gradients/add_grad/Sum\n",
      "gradients/add_grad/Reshape\n",
      "gradients/add_grad/Sum_1\n",
      "gradients/add_grad/Reshape_1\n",
      "gradients/add_grad/tuple/group_deps\n",
      "gradients/add_grad/tuple/control_dependency\n",
      "gradients/add_grad/tuple/control_dependency_1\n",
      "gradients/MatMul_grad/MatMul\n",
      "gradients/MatMul_grad/MatMul_1\n",
      "gradients/MatMul_grad/tuple/group_deps\n",
      "gradients/MatMul_grad/tuple/control_dependency\n",
      "gradients/MatMul_grad/tuple/control_dependency_1\n",
      "beta1_power/initial_value\n",
      "beta1_power\n",
      "beta1_power/Assign\n",
      "beta1_power/read\n",
      "beta2_power/initial_value\n",
      "beta2_power\n",
      "beta2_power/Assign\n",
      "beta2_power/read\n",
      "W/Adam-op/Initializer/zeros\n",
      "W/Adam-op\n",
      "W/Adam-op/Assign\n",
      "W/Adam-op/read\n",
      "W/Adam-op_1/Initializer/zeros\n",
      "W/Adam-op_1\n",
      "W/Adam-op_1/Assign\n",
      "W/Adam-op_1/read\n",
      "b/Adam-op/Initializer/zeros\n",
      "b/Adam-op\n",
      "b/Adam-op/Assign\n",
      "b/Adam-op/read\n",
      "b/Adam-op_1/Initializer/zeros\n",
      "b/Adam-op_1\n",
      "b/Adam-op_1/Assign\n",
      "b/Adam-op_1/read\n",
      "Adam-op/learning_rate\n",
      "Adam-op/beta1\n",
      "Adam-op/beta2\n",
      "Adam-op/epsilon\n",
      "Adam-op/update_W/ApplyAdam\n",
      "Adam-op/update_b/ApplyAdam\n",
      "Adam-op/mul\n",
      "Adam-op/Assign\n",
      "Adam-op/mul_1\n",
      "Adam-op/Assign_1\n",
      "Adam-op\n",
      "ArgMax/dimension\n",
      "ArgMax\n",
      "ArgMax_1/dimension\n",
      "ArgMax_1\n",
      "correct_pred\n",
      "Cast\n",
      "Const_2\n",
      "accuracy\n",
      "save/Const\n",
      "save/SaveV2/tensor_names\n",
      "save/SaveV2/shape_and_slices\n",
      "save/SaveV2\n",
      "save/control_dependency\n",
      "save/RestoreV2/tensor_names\n",
      "save/RestoreV2/shape_and_slices\n",
      "save/RestoreV2\n",
      "save/Assign\n",
      "save/RestoreV2_1/tensor_names\n",
      "save/RestoreV2_1/shape_and_slices\n",
      "save/RestoreV2_1\n",
      "save/Assign_1\n",
      "save/RestoreV2_2/tensor_names\n",
      "save/RestoreV2_2/shape_and_slices\n",
      "save/RestoreV2_2\n",
      "save/Assign_2\n",
      "save/RestoreV2_3/tensor_names\n",
      "save/RestoreV2_3/shape_and_slices\n",
      "save/RestoreV2_3\n",
      "save/Assign_3\n",
      "save/RestoreV2_4/tensor_names\n",
      "save/RestoreV2_4/shape_and_slices\n",
      "save/RestoreV2_4\n",
      "save/Assign_4\n",
      "save/RestoreV2_5/tensor_names\n",
      "save/RestoreV2_5/shape_and_slices\n",
      "save/RestoreV2_5\n",
      "save/Assign_5\n",
      "save/RestoreV2_6/tensor_names\n",
      "save/RestoreV2_6/shape_and_slices\n",
      "save/RestoreV2_6\n",
      "save/Assign_6\n",
      "save/RestoreV2_7/tensor_names\n",
      "save/RestoreV2_7/shape_and_slices\n",
      "save/RestoreV2_7\n",
      "save/Assign_7\n",
      "save/restore_all\n",
      "init\n"
     ]
    }
   ],
   "source": [
    "# delete the current graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# import the graph from the file\n",
    "imported_graph = tf.train.import_meta_graph('linear_model.meta')\n",
    "\n",
    "# list all the tensors in the graph\n",
    "for tensor in tf.get_default_graph().get_operations():\n",
    "    print (tensor.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say that I am interested in `loss` and `accuracy` of my model. We can easily get the values of corresponding tensors, providing the correct placeholders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./linear_model\n",
      "---------------------------------------------------------\n",
      "Test loss: 0.32, test accuracy: 91.2%\n",
      "---------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# run the session\n",
    "with tf.Session() as sess:\n",
    "    # restore the saved vairable\n",
    "    imported_graph.restore(sess, './linear_model')\n",
    "        \n",
    "    # Accuracy\n",
    "    feed_dict_test = {'X:0': data.test.images, 'Y:0': data.test.labels}\n",
    "    loss_test, acc_test = sess.run(['loss:0', 'accuracy:0'], feed_dict=feed_dict_test)\n",
    "    print('---------------------------------------------------------')\n",
    "    print(\"Test loss: {0:.2f}, test accuracy: {1:.01%}\".format(loss_test, acc_test))\n",
    "    print('---------------------------------------------------------')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for reading! If you have any question or doubt, feel free to leave a comment in our [website](http://easy-tensorflow.com/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
